Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating sql connector version and testing integration with OAuth #436

Merged
merged 6 commits into from
Sep 7, 2023

Conversation

benc-db
Copy link
Collaborator

@benc-db benc-db commented Sep 5, 2023

Initially aimed to resolve #433 and #410, but now believe there was a different issue at play there.

Description

  • Updates the databricks-sql-connector version. Initially I updated the version to >=2.9.0, but upon further testing, I didn't actually find an issue with 2.7.0. However, our previous version spec did not allow 2.9.x, which seems like an oversight.

  • Updates our integration test suite on Github Actions to use OAuth rather than token. As we are advertising that the connector supports this feature and that it is safer than using a PAT, we gain more by dogfooding with OAuth than continuing to test only PAT.

  • Updates the workflow README to inform users that jobs clusters are no longer supported for dbt tasks in Databricks Workflows (and haven't been for several months at least).

Checklist

  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

@benc-db benc-db temporarily deployed to azure-prod September 5, 2023 21:14 — with GitHub Actions Inactive
@benc-db benc-db temporarily deployed to azure-prod September 5, 2023 21:14 — with GitHub Actions Inactive
@benc-db benc-db temporarily deployed to azure-prod September 5, 2023 21:28 — with GitHub Actions Inactive
@benc-db benc-db temporarily deployed to azure-prod September 5, 2023 21:28 — with GitHub Actions Inactive
@benc-db benc-db temporarily deployed to azure-prod September 5, 2023 21:31 — with GitHub Actions Inactive
@benc-db benc-db temporarily deployed to azure-prod September 5, 2023 21:31 — with GitHub Actions Inactive
@benc-db benc-db temporarily deployed to azure-prod September 5, 2023 21:36 — with GitHub Actions Inactive
@benc-db benc-db temporarily deployed to azure-prod September 5, 2023 21:36 — with GitHub Actions Inactive
@benc-db benc-db marked this pull request as ready for review September 5, 2023 21:36
susodapop
susodapop previously approved these changes Sep 6, 2023
requirements.txt Outdated
@@ -1,4 +1,4 @@
databricks-sql-connector~=2.7.0
databricks-sql-connector>=2.7.0

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason not to simply pin to the latest connector version? I'm okay with a wider specification, but it might be nicer to just hard-code to the latest and greatest.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only issue is keeping them in sync. Per our discussion this morning, I'm ok updating to >= 2.9.3, <3. If you make optional improvements in databricks-sql-connector, I don't want to force this project to have a release just to get access to them.

@benc-db benc-db dismissed stale reviews from susodapop and rcypher-databricks via 8da0065 September 6, 2023 22:29
@benc-db benc-db force-pushed the sql-connector-version branch from dd9ed06 to 8da0065 Compare September 6, 2023 22:29
@benc-db benc-db temporarily deployed to azure-prod September 6, 2023 22:29 — with GitHub Actions Inactive
@benc-db benc-db temporarily deployed to azure-prod September 6, 2023 22:29 — with GitHub Actions Inactive
@benc-db benc-db temporarily deployed to azure-prod September 6, 2023 22:32 — with GitHub Actions Inactive
@benc-db benc-db temporarily deployed to azure-prod September 6, 2023 22:32 — with GitHub Actions Inactive
@benc-db benc-db temporarily deployed to azure-prod September 6, 2023 23:10 — with GitHub Actions Inactive
@benc-db benc-db temporarily deployed to azure-prod September 6, 2023 23:10 — with GitHub Actions Inactive
@benc-db benc-db merged commit de2d44f into main Sep 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ServicePrinicipal Authentication (OAuth) doesn't work while running dbt jobs on Databricks
3 participants